Setting Up AI Cluster
Create a Lossless Fabric for the AI workloads
After installing Cisco NDFC, you decide to test NDFC automation capabilities by deploying BGP EVPN fabric if the company decides to expand AI workloads across several sites in the future.
In this task, you will deploy the EBGP fabric with configured AI/ML QoS and queuing policies. This way, you will achieve lossless behavior in the network, which is required for the AI workload communication over RoCEv2. You will use Cisco NDFC templates to automate the deployment of fabric and QoS configuration.
Step 1
On your desktop, locate and open the Chrome web browser. Then, navigate to https://10.0.0.70.
Answer
You should see the login page for the Cisco Nexus Dashboard.
Accept any alerts regarding privacy by first clicking Advanced and then clicking Proceed to 10.0.0.70 (unsafe), as shown in the figures.
Step 2
Enter admin and C!sco123 in the username and password fields, respectively. Click the Login button.
Answer
You will see the Welcome, admin page with the Journey tab selected. The Getting Started Map is partially complete and it shows the next step, which is adding a fabric.
If you see an alert regarding privacy concerns, click Advanced, followed by clicking Proceed to 10.0.0.70 (unsafe), as depicted in the figures.
Step 3
Click Admin Console in the top bar and choose Fabric Controller from the menu.
Answer
You will see the Overview page of the Cisco Nexus Dashboard Fabric Controller.
Step 4
To manage the NDFC fabrics, hover over the Manage section in the left navigation panel and choose Fabrics.
Answer
Notice that the widgets show no information because you have not created fabrics or added switches to the Cisco Nexus Dashboard.
Step 5
Click the Actions button, and choose Create Fabric from the drop-down list.
Answer
Step 6
Name the fabric AI-Fabric, and then click the Choose Fabric button.
Answer
Step 7
Click the BGP Fabric template to highlight it, and then click the Select button.
Answer
The NDFC templates provide an already populated configuration to easily deploy complex fabrics across your network devices. In this task, you will use the BGP Fabric template to deploy the EBGP underlay with QoS policies to ensure lossless traffic.
Step 8
In the General Parameters tab, enter 65000 in the BGP ASN for Spines field. Verify that Multi-AS is chosen from the BGP AS Mode drop-down list, and check the Allow Same ASN On Leafs option.
Answer
Even though the Multi-AS setting allows you to use a unique ASN for each leaf, you also checked the option Allow Same ASN On Leafs. This option lets you configure the same BGP ASN on leafs. The same ASN on leaf switches is typically used when you want to create a single virtual port channel (vPC) pair.
Step 9
Click the EVPN tab to inspect the configuration. Make sure that the Enable EVPN VXLAN Overlay, Enable VXLAN OAM, and Enable Tenant DHCP options are enabled.
Answer
You should see that the EVPN overlay settings are already enabled by the BGP Fabric template.
Step 10
Click the Protocols tab, and check the Enable BFD and Enable BFD for BGP check boxes.
Answer
The Bidirectional Forwarding Detection (BFD) option enables faster convergence times when the fabric is being created.
Step 11
Click the Advanced tab. Scroll down and choose Enable from the Greenfield Cleanup Option drop-down list.
Answer
The Greenfield Cleanup Option allows you to clean the switch configuration when adding it to the NDFC without a switch reload. This option is recommended only for environments with virtual switches such as the Cisco Nexus 9000v that you use in this task.
Step 12
Scroll further down and check the Enable AI/ML QoS and Queuing Policies check box. Choose AI_Fabric_QOS_25G from the AI/ML QoS & Queuing Policy drop-down list.
Answer
The AI_Fabric_QOS_25G template provides the QoS configuration that will be applied to the entire fabric, including PFC and ECN configuration. The PFC and ECN congestion mechanisms need to be enabled in tandem to ensure lossless communication across the fabric. You will inspect the QoS configuration in closer detail when you deploy the configuration to the switches.
Note
In this scenario, you selected the 25G policy, but typically you would select the template that reflects the speed of the physical ports on your switches.
Step 13
Open the terminal by pressing Ctrl+Alt+T or by clicking the Terminal icon in the bottom taskbar on the jump host desktop.
Answer
You will see the terminal window.
Step 14
Type cat freeform.conf in the terminal and press the Enter key.
Answer
Your terminal will now display the contents of the freeform.conf file.
Step 15
Select the lines with the content, right-click, and choose Copy from the menu to copy the contents to the clipboard.
Answer
Step 16
Back in your browser, scroll to the bottom, and paste the freeform configuration you have in the clipboard to the Intra-fabric Links Additional Config.
Answer
The freeform configuration window allows you to specify the commands that will be applied to intra-fabric links in addition to the configuration provided by the NDFC templates. These commands will enable PFC and QoS traffic classification on the interfaces between spine and leaf switches. The priority-flow-control watch-dog-interval on command enables the watch-dog timer that will monitor PFC frames and prevent potential PFC storms in the fabric.
Step 17
Click the Save button in the lower-right corner of the page to save the fabric configuration.
Answer
You will land back on the Fabrics page that will now display the AI-Fabric that you have just configured.
Add Switches to the Fabric
Impressed with how easy it is to configure the BGP fabric with Cisco NDFC fabric templates, you decide to add your Cisco Nexus switches to the BGP fabric. But before pushing the configuration to the switches, you want to verify what kind of QoS configuration is NDFC deploying.
Step 18
In the bottom toolbar, locate and click the terminal emulator icon.
Answer
You will begin by connecting to the switch and checking the QoS configuration.
Step 19
Use the ssh admin@172.16.0.10 command to connect to the leaf01 switch. If prompted to continue connecting, enter yes and use C!sco123 as the password.
Answer
You should get an output with the licensing information and a prompt that is set to the leaf01 switch. You will see that Cisco Nexus 9000v virtual switches are used in this task.
Step 20
Use the show run ipqos command to check if the switch has the QoS configuration applied to it.
Answer
You should see empty output. You can verify other switches using the same method. At this point, all switches should have no QoS configuration applied to it. In the following steps, you will use Cisco NDFC to apply the configuration by simply adding switches to the AI fabric that you created in the previous task.
Step 21
Navigate back to the browser where you have Cisco NDFC opened. Double-click the AI-Fabric to open Fabric Overview.
Answer
You should see that the fabric is healthy, but has no switches added to it.
Step 22
In the top-right corner, click the Actions button, and choose Add Switches from the drop-down list.
Answer
Step 23
Enter the IP range 172.16.0.10-20 in the Seed IP field, admin as the username, and C!sco123 as the password. Verify that the Preserve Config check box is unchecked.
Answer
The provided IP range should discover both leafs and spine. You can always access their information in the lab job aid at the beginning of the lab guide. Cisco NDFC will automatically discover all devices within the seed range and devices that are within the defined Max Hops range from the seed switches. The Preserve Config check box allows you to add the switch to the fabric without losing any configuration. When the box is not checked, the switch configuration is cleared during the import process, which removes all configuration except the management details and reboots the switch.
Step 24
Click the Discover Switches button in the bottom-right corner.
Answer
Step 25
Click Confirm in the warning alert to proceed.
Answer
After a few seconds, you should see the discovered devices in the Discovery Results table. Specifically, you should see leaf01, leaf02 and spine01 discovered and their status reported as Manageable. Notice that NDFC pings all IP addresses within the provided range and reports the details about the device such as Serial Number and Model.
Step 26
Click the check box next to the Switch Name to choose the leaf01, leaf02, and spine01 switches. Then click the Add Switches button in the lower-right corner.
Answer
In the process of importing switches, Cisco NDFC erases the configuration on the switches and reboots it. The import process can take up to 10 minutes with physical switches, but with Cisco Nexus 9300v virtual switches the process will be finished within a few minutes.
Once the import process is complete, the switch Status changes to Switch Added, and the Progress bar changes to Green.
Step 27
Close the window by clicking X in the upper-right corner.
Answer
You should land back on the Fabric Overview window.
Step 28
Navigate to the Switches tab, and verify that all three switches are now added to the fabric.
Answer
If the Discovery Status column reports Unreachable, choose the unreachable switches, click the Actions button, select Discovery, and click the Rediscover button. By doing rediscovery, Cisco NDFC attempts to reconnect to the switch and obtain switch details. The Discovery Status on all three switches should be OK.
Note
The Operational Status column might indicate Minor alert status because some interfaces are initialized as down. This status has no effect on the lab task.
Step 29
Choose the spine01 switch, click the Actions button, and choose Set Role from the drop-down list.
Answer
By default, NDFC will assign leaf role to all newly added switches, including the spine switches. Spine or any other device roles are assigned manually by selecting them and changing the role. You can select one or multiple devices at once.
Step 30
In the Select Role pop-up window, click Spine to highlight the choice. Then click the Select button in the lower-right corner of the pop-up window.
Answer
After selecting the role, you should see the Warning window advising you to trigger the Recalculate and Deploy action.
Click OK to close it. The spine01 switch is now properly assigned with the spine role.
Step 31
Click the upper, dark blue Actions button that is located beside the Refresh icon, and choose Recalculate and Deploy from the drop-down list.
Answer
The Recalculate and Deploy action will start calculating the configuration that will be applied to the devices.
After the window with calculation progress closes, you should see the Deploy Configuration window.
The Fabric Status column will report the Out-of-Sync status, indicating you have pending configuration that can be deployed to the added devices. The Pending Config column shows how many lines of the configuration are pending to be deployed.
Note
The order of the devices that you see in the table might be different compared to the lab guide.
Step 32
Click the blue link under the Pending Config column in the leaf01 row to check which commands will be applied to leaf01.
Answer
After scrolling down, you should see the QoS configuration that will be applied to leaf01. This configuration is provided by the AI/ML QoS template that you included in the BGP Fabric template.
In the configuration below, the class-map type qos command ensures that all traffic marked with DSCP 26 will be classified as RoCEv2 traffic. Also, all traffic marked with a DSCP value of 48 will be marked as CNP (Congestion Notification Packet). The goal is to classify the RoCEv2 and CNP traffic, so that switches can put it in the right queue.
class-map type qos match-any ROCEv2 match dscp 26 class-map type qos match-any CNP match dscp 48
The policy-map configuration classifies appropriate traffic to the right queue. In this example, RoCEv2 traffic is classified to queue 3, and CNP control traffic is classified to the strict-priority queue 7 to deliver congestion notification packets as quickly as possible.
policy-map type qos QOS_CLASSIFICATION
class ROCEv2
set qos-group 3
class CNP
set qos-group 7
class class-default
set qos-group 0After traffic classification, you should see the queuing and scheduling configuration. The queue 3 (c-out-8q-q3) is assigned 50% of the bandwidth. The minimum threshold is set to 100 KB, the maximum threshold to 700 KB, and if there is congestion happening, mark the packets with ECN. The queue 7 (c-out-8q-q7) has priority 1 assigned to it. This queue handles the CNP packets.
policy-map type queuing QOS_EGRESS_PORT
class type queuing c-out-8q-q6
bandwidth remaining percent 0
class type queuing c-out-8q-q5
bandwidth remaining percent 0
class type queuing c-out-8q-q4
bandwidth remaining percent 0
class type queuing c-out-8q-q3
bandwidth remaining percent 50
random-detect minimum-threshold 100 kbytes maximum-threshold 700 kbytes drop-probability 40 weight 0 ecn
class type queuing c-out-8q-q2
bandwidth remaining percent 0
class type queuing c-out-8q-q1
bandwidth remaining percent 0
class type queuing c-out-8q-q-default
bandwidth remaining percent 50
class type queuing c-out-8q-q7
priority level 1To enable the PFC pause frames for a specific queue, an additional QoS configuration should be applied. In this example, the PFC pause frames are enabled on queue 3 and will be generated if congestion happens.
policy-map type network-qos qos_network
class type network-qos c-8q-nq3
pause pfc-cos 3
mtu 4200The service-policy commands are required to apply queuing and QoS configurations systemwide. So, if there is congestion, packets will be marked with ECN. The ports with PFC will receive those frames and send pause frames if congestion thresholds are triggered.
system qos service-policy type network-qos qos_network service-policy type queuing output QOS_EGRESS_PORT
Scroll further down to verify the configuration of interface Ethernet1/1. To implement end-to-end lossless fabric, PFC must be enabled on all interfaces across the network. The RoCEv2 traffic classification is enabled at the interface level, including the logical Network Virtual Interface (nve1) where traffic encapsulation and de-encapsulation occurs.
interface ethernet1/1 priority-flow-control mode on priority-flow-control watch-dog-interval on service-policy type qos input QOS_CLASSIFICATION no switchport ip address 10.4.0.2/30 description connected-to-spine01-Ethernet1/1 mtu 9216 ip pim sparse-mode no shutdown interface nve1 source-interface loopback1 host-reachability protocol bgp service-policy type qos input QOS_CLASSIFICATION
Step 33
Click the Close button in the bottom-right corner to close the Pending Config window.
Answer
Step 34
Click the Deploy All button in the bottom-right corner of the Deploy Configuration - AI-Fabric window.
Answer
Configuration deployment will take a couple of minutes. The Progress column will provide you with progress information. Once the changes are deployed, you should see the Success status in the Progress column.
Step 35
Click the Close button to return to the Fabric Overview window.
Answer
Newly added switches should now have the In-Sync Config Status.
Step 36
Select the leaf01 switch, click the Actions button to open the drop-down list, hover over More, and click Show Commands.
Answer
Step 37
From the Commands drop-down list, choose the show command.
Answer
Note
Once switches are added to Cisco NDFC, you can still use the switch CLI to run commands. In this example, Cisco NDFC is used to run commands on the switch.
Step 38
Under Show Command 1, enter the run ipqos command, and click Execute in the bottom-right corner to verify if the QoS configuration was applied to leaf01.
Answer
The Switch show commands window allows you to run various show commands and verify the switch running configuration. Notice that the QoS configuration is now applied to the switch.
Step 39
Click the X button in the top-right corner to close the window.
Answer
Configure the Overlay Network
After deploying the BGP fabric on the switches and inspecting the AI/ML QoS policy configuration, you are gaining confidence in the automation capabilities of Cisco NDFC. You decide to also configure the EBGP overlay network to distribute reachability information about the endpoints. You will continue to use Cisco NDFC policies to configure leaf BGP ASNs and add your spine switches as EBGP neighbors to your leaf switches and the other way round.
Step 40
In the Fabric Overview window, click the Policies tab.
Answer
You should see the list of all policies that are applied to the fabric. The policies provide an easy way to deploy or change configuration across the fabric.
Step 41
Click the white Actions button, and choose Add Policy from the drop-down list.
Answer
Step 42
Choose the leaf01 and leaf02 switch by clicking the check boxes next to their names, and click Next in the bottom-right corner.
Answer
Make sure you choose only leaf switches, because you will deploy the policy that will configure BGP ASNs on the leafs.
Step 43
In the Create Policy window, click the No Policy Selected button.
Answer
The Policy Template window should appear with the list of all defined policy templates in Cisco NDFC.
Step 44
Type asn in the search bar to find the leaf_bgp_asn policy. Click the policy to highlight it and click Select.
Answer
You should see the Create Policy window after selecting the policy.
Notice that now the Create Policy window shows the Leaf BGP AS # field.
Step 45
In the Leaf BGP AS # field, enter 65010, leave all other parameters at their default values, and click the Save button in the bottom-right corner.
Answer
After a few seconds, you should be redirected to the Policies tab and see a Success pop-up window in the bottom-right corner.
Step 46
Under the Policies tab, click the Filter by attributes bar to open a drop-down filter list, and select Template.
Answer
After selecting Template, the filter will immediately provide another drop-down list.
Note
The filter field provides suggestions when typing and supports the autocomplete command when pressing the Tab key on the keyboard.
Step 47
Choose contains from the drop-down list, and then type asn to complete the filter. Press Enter to search.
Answer
The filter field provides suggestions when typing and supports the autocomplete command when pressing the Tab key on the keyboard.
After performing the search, among the filtered policies, you will find the leaf_bgp_asn and bgp_asn policies that are created for each leaf switch. Notice that even though you configured only the leaf_bgp_asn policy, an additional bgp_asn policy was created for both leafs.
To configure the policies for EBGP neighboring between the leafs and spine, first you need to find out the IP addresses of the Loopback0 interfaces. The NDFC BGP template configured the loopback interfaces when you added switches to the fabric. You need to collect the IPs.
Step 48
Navigate to the Interfaces tab in the Fabric Overview window, click the Filter by attributes bar, and choose Interface.
Answer
Step 49
Click the == filter from the drop-down list, type Loopback0 and press Enter to search for the interfaces.
Answer
Step 50
Move the horizontal scroll bar to the right to see the IP/Prefix column and collect the IPs.
Answer
You should get similar outputs as shown below. In this example, the leaf01 Lo0 interface has IP address 10.2.0.1, the leaf02 Lo0 interface has IP address 10.2.0.3, and the spine01 Lo0 interface has IP address 10.2.0.2.
Note
Cisco NDFC assigns the Lo0 IP addresses when you add switches to the fabric, and your IPs might be different than shown in this example.
Step 51
In the Fabrics Overview page, choose the Policies tab. Click the white Actions button and choose Add Policy from the drop-down list.
Answer
Step 52
Choose the leaf01 and leaf02 switches and click Next in the bottom-right corner.
Answer
Step 53
Click the No Policy Selected button to select the policy template.
Answer
Step 54
In the search bar, type ebgp_overlay, click the ebpg_overlay_leaf_all_neighbor policy to highlight it, and click the Select button.
Answer
You should be back at the Create Policy page. Notice that you have two new fields to populate.
Step 55
In the Spine/Super Spine IP List field, enter the IP address that you have collected from the Lo0 interface on the spine01 switch. The BGP Update Source Interface by default should have the Loopback0 interface selected. Click Save in the bottom-right corner.
Answer
In this example, the IP is 10.2.0.2, but in your environment spine01 might have different IPs. Make sure to enter the IP that you have previously collected from the Interfaces tab.
After a couple of seconds, you should land back to the Policies tab and see a notification that the policies were successfully added.
Now you need to create a similar policy for the spine01 switch.
Step 56
From the Policies tab, click the white Actions button and choose Add Policy from the drop-down list.
Answer
Step 57
Check the check box next to the spine01 switch and click the Next button.
Answer
Step 58
Click the No Policy Selected button to select the policy template.
Answer
The Select Policy Template window should appear.
Step 59
In the search bar, type ebgp_overlay, click the ebpg_overlay_spine_all_neighbor policy to highlight it, and click the Select button.
Answer
In the Create Policy page, you should now see three additional fields to populate.
Step 60
In the Leaf IP List field, enter the IP addresses that you have collected from the Lo0 interfaces on the leaf01 and leaf02 switch. In the Leaf BGP ASN field, enter both leaf ASN numbers in a comma-separated list: 65010, 65010. The BGP Update Source Interface by default should have the Loopback0 interface selected. Click Save in the bottom-right corner.
Answer
In this example, leaf Lo0 IPs are 10.2.0.1 and 10.2.0.3, but in your environment, interfaces might have different IPs. Make sure to enter the Lo0 IPs that you have collected from the Interfaces tab.
After a couple of seconds, you should land back to the Policies tab and see a notification that the policies were successfully added.
After you configured the policies, you will need to deploy these changes to the switches. For any kind of configuration changes you make on fabric or policy level, Cisco NDFC must calculate the appropriate set of commands to reflect those changes.
Step 61
Click the blue Actions button in the top-right corner, and choose Recalculate and Deploy from the drop-down list.
Answer
You should see the recalculation progress in the temporary pop-up window.
After the calculation is done, you should be forwarded to the Deploy Configuration window. Because there are pending configuration changes on leaf01, leaf02 and spine01, Cisco NDFC is reporting the Fabric Status as Out-of-Sync on all three devices. If you want to inspect the pending BGP configuration, you can click the blue link under the Pending Config column for each device.
Step 62
Click Deploy All in bottom-right corner to deploy the configuration.
Answer
You should see the progress of the configuration deployment.
It will take Cisco NDFC a few seconds to deploy the configuration. When the configuration is deployed, you should see the Deployment completed status.
Step 63
Click the X button in the top-right corner to close the window.
Answer
You will return to the Fabric Overview page with the Policies tab selected. The spines and leafs now have a functioning EBGP overlay configured.
Configure the Network for AI Workloads
Having EBGP overlay configured and operational, you start thinking how to connect the AI cluster to the fabric. Because the AI cluster will generate a lot of traffic and process many sensitive documents, you decide to create a dedicated VRF and AI network subnet. This way, you will segregate the AI cluster from other workloads. You will also attach the AI network to the switch interfaces, so that endpoints Student VM and PC-01 get access to the network. Finally, by testing communication from Student VM (attached to Eth1/3 on leaf01) to PC-01 (attached to Eth1/3 on leaf02), you will test the end-to-end connectivity to confirm if the network is configured properly.
Note
The lab environment is using Cisco Nexus 9000v virtual switches. These switches allow you to configure QoS and congestion mechanisms for the AI network just like you would on the physical Cisco Nexus 9000 switches. Nonetheless, virtual Cisco Nexus 9000v switches cannot implement QoS policies and apply congestion mechanisms when congestion happens, that is only possible on physical Cisco Nexus switches. Therefore, within this lab exercise you will test the network connectivity for the configured network.
Step 64
In the Fabric Overview window, click the VRFs tab. Click the white Actions button and choose Create from the drop-down list.
Answer
You should see a new Create VRF window.
Notice that Cisco NDFC will automatically select the default VRF template and provide some default values for the VRF configuration.
Step 65
Change the VRF Name to AI_VRF_50000. Click the Propose VLAN button to allow Cisco NDFC to suggest the VLAN ID from the pool, and leave all other fields with the default values. Click Create in the bottom-right corner.
Answer
Because you are deploying this VRF for AI workloads, this task suggests the AI_VRF_50000 name, but you can choose any VRF name that you want.
After clicking Create, you will be redirected back to the VRFs tab, which is now showing the VRF that you have created. You should see the VRF status as N/A because it is not yet deployed on the switches.
Step 66
Click the Networks tab, then click the white Actions button and choose Create from the drop-down list.
Answer
You should see a new Create Network window. Notice that Cisco NDFC already selected the default Network template with the default values populated in the provided fields to speed up the configuration.
Step 67
Change the Network Name to AI_Network_30000 and click the Propose VLAN ID button to get the available VLAN ID from the pool. Type 192.168.1.254/24 in the IPv4 Gateway/NetMask field and AI_Workloads in the VLAN Name field. To confirm network creation, click the Create button.
Answer
Because you are creating a dedicated network and VLAN ID for the AI workloads, this example suggests using the AI_ prefix for the network and VLAN name. You can choose other names, but it is good practice to define descriptive names that allow you to easily identify their purpose. Also notice the IP of the gateway that you defined—it is the default gateway your AI workloads will use to access the network.
After clicking Create, you will be redirected back to the Networks tab.
You should see your network for AI workloads in the Networks tab. The Network Status is reported as N/A because the network is not yet assigned to the interfaces and deployed.
Step 68
At the bottom of the screen, locate and click the terminal emulator icon, and type the ip a command to list the network interfaces of Student VM.
Answer
The ip a command will list all interfaces on the Student VM. Notice four interfaces. One of them is the ens224 interface, which is configured with IP address 192.168.1.1/24. This IP address is from the same subnet as your AI network.
Step 69
Type the ip r command to list the routing table of Student VM.
Answer
Notice that the default gateway IP address 192.168.1.254 is already configured. This gateway IP address is the same that you configured for your AI network.
Step 70
Type ping 192.168.1.254, press Enter to check the reachability of the default gateway, and after 10 seconds press Ctrl+C to stop the pings. You will go back to the Cisco NDFC and leave the terminal window open.
Answer
You should see the Destination Host Unreachable message for each ping and 100% packet loss because the network is not yet deployed on leaf interfaces.
Step 71
At the bottom of the screen, locate and click the browser icon to return to your Cisco NDFC tab.
Answer
You should land back on the Fabric Overview Networks tab.
Step 72
In the Fabric Overview window, click the Interfaces tab to see all physical and virtual interfaces on all the switches in the fabric that Cisco NDFC discovered.
Answer
You should see the various attributes of the interfaces. Without applying a filter, it might be difficult to find specific information or interface because this view lists all interfaces in the fabric.
Step 73
Click Filter by attributes. Create the Interface == Ethernet1/3 filter and press Enter to filter interfaces.
Answer
You should see three interfaces after applying the filter. The interfaces will be Down because they are not yet configured or attached to the network.
In the following steps, you will configure the Ethernet1/3 interfaces on leaf01 and leaf02 as access interfaces. You will attach them to the AI network, so that Student VM (connected to Ethernet1/3 on leaf01) and PC-01 (connected to Ethernet1/3 on leaf02) can communicate over the 192.168.1.0/24 AI network.
Step 74
Select the Ethernet1/3 interfaces on the leaf01 and leaf02 switches, click the white Actions button, and choose Edit from the drop-down list.
Answer
Step 75
Click the int_trunk_host string under the Policy field to change the policy template.
Answer
You should see the Select Attached Policy Template window.
By default, Cisco NDFC is using the int_trunk_host policy for this interface. You will reconfigure this interface as access interface by choosing a different interface template.
Note
Notice that the Edit Interfaces window shows you are editing 1 of 2 Selected Interfaces, specifically you are editing the leaf01 Ethernet1/3 interface. After you are done with editing this interface, you will repeat the process for the leaf02 Ethernet1/3 interface.
Step 76
Type access in the search bar, click the int_access_host policy template to select it, and click the Select button when you are done.
Answer
After clicking the Select button, you should land back on the Edit Interfaces window. Notice the different configuration options available after you have changed interface policy from trunk to access type.
Step 77
In the Access Vlan field, enter 2300. In the Interface Description field, type Link to Student VM or something equally descriptive.
Answer
Make sure that the Access Vlan value is the same ID value as you specified for your AI network VLAN ID.
Step 78
Scroll further down to make sure that the Enable Interface, Enable Priority Flow Control, and Enable QoS Configuration check boxes are checked. Click the Save & Next button in the bottom-right corner.
Answer
The PFC and QoS configuration needs to be applied end-to-end to have lossless fabric. This includes configuring QoS and PFC on access interfaces as shown in this example. You have already configured the spine-to-leaf connections in the previous tasks when you configured the BGP fabric template.
After clicking the Save & Next button, you will repeat the same configuration steps, but this time for the Ethernet1/3 interface of the leaf02 switch.
Step 79
Click the int_trunk_host string under the Policy field to change the policy template.
Answer
Notice that you are now editing the Ethernet1/3 interface of the leaf02 switch. By default, Cisco NDFC is using the int_trunk_host policy for this interface. You will reconfigure this interface as access interface by selecting a different interface template.
Step 80
Type access in the search bar, click the int_access_host policy template to select it, and click the Select button when you are done.
Answer
Step 81
In the Access Vlan field, enter 2300. In the Interface Description field, type Link to PC-01 or something equally descriptive.
Answer
The configuration process for the Ethernet1/3 interface of the leaf02 switch will be very similar. Make sure that the Access Vlan value is the same ID value as you specified for your AI network VLAN ID and the leaf01 Ethernet1/3 interface. The description is now Link to PC-01, because it is the interface where PC-01 is connected.
Step 82
Scroll further down to make sure that the Enable Interface, Enable Priority Flow Control, and Enable QoS Configuration check boxes are checked. In the bottom-right corner, click the Save button first, and then click Deploy.
Answer
The configuration process for the Ethernet1/3 interface of the leaf02 switch will be very similar. Make sure that the Access Vlan value is the same ID value as you specified for your AI network VLAN ID. The description is now Link to PC-01, because it is the interface where PC-01 is connected.
After you click the Save button, you will get a notification in the bottom-right corner that you successfully edited the interfaces. Also, notice that the Deploy button is now clickable.
After clicking the Deploy button, you will be forwarded to the Deploy interfaces configuration screen showing you the Pending Configuration for the Ethernet1/3 interfaces.
Step 83
Click the Pending Config link for one of the interfaces to inspect the configuration.
Answer
Notice that NDFC is enabling PFC and QoS classification of the traffic on the Ethernet1/3 interface. This way, RoCEv2 traffic is immediately classified when it enters the fabric and the access interface will be able to use the PFC mechanism to resolve congestion if required.
Step 84
Click the X icon in the top-right corner to close the Pending Config window.
Answer
Step 85
In the bottom-right corner, click the Deploy Config button to deploy the configuration.
Answer
After a few seconds, you should see the Deploy interfaces configuration window, showing the progress of the deployment.
After deployment is done, NDFC will automatically return you the Edit Interfaces window. In the bottom-right corner, you should notice a notification about successful configuration deployment.
Step 86
Click the X button in the top-right corner to exit the Edit Interfaces window.
Answer
Notice that the interfaces now show Admin Status value UP. However, the Operational Status is still Down because you still didn't attach interfaces to the network.
Note
If the interface page remains the same for more than a couple of seconds, reload the page to refresh the view.
Step 87
Choose again both Ethernet1/3 interfaces on leaf switches, click the white Actions button, and then choose Multi-Attach from the drop-down list.
Answer
You should land on the Interface Multi-Attach window, where you can attach interfaces to the available networks. You should only see AI_Network_30000, which you configured for AI workloads.
Step 88
Choose AI_Network_30000 as the network where you will attach the selected Ethernet1/3 interfaces and click Next in the bottom-right corner.
Answer
Cisco NDFC will forward you to the Summary step where you can verify interface association.
Step 89
Make sure that the Proceed to Full Switch Deploy option is chosen, and click the Save button in the bottom-right corner.
Answer
After clicking the Save button, you should see the Deploy Configuration window. Both leaf switches have pending configuration.
Step 90
Click the Deploy All button to confirm deployment.
Answer
The deployment of this configuration takes a few seconds. When it is done, you should see the Deployment Completed status.
Step 91
Click the X button in the top-right corner to close the window.
Answer
Cisco NDFC will return you to the Interfaces tab. In this tab, you should notice that both Ethernet1/3 interfaces are now reporting Admin and Operational Status as UP.
Step 92
Click the terminal icon in the bottom toolbar, type the ping 192.168.1.254 command and press Enter to verify if Student VM can ping the gateway.
Answer
Notice that now the ping command works and that Student VM can reach the gateway.
Step 93
Use the ping 192.168.1.103 command to verify that Student VM can also reach PC-01, which is attached to the Ethernet1/3 interface on the leaf02 switch.
Answer
Notice that Student VM can also ping PC-01, which is connected to the other leaf switch.
With this task, you verified the connectivity between PC-01 and Student VM, which are communicating over a lossless network.
Note
The lab environment is using Cisco Nexus 9000v virtual switches. These switches allow you to configure QoS and congestion mechanisms for the AI network just like you would on the physical Cisco Nexus 9000 switches. Nonetheless, virtual Cisco Nexus 9000v switches cannot implement QoS policies and apply congestion mechanisms when congestion happens, that is only possible on physical Cisco Nexus switches. Therefore, this lab exercise does not cover congestion simulation.